1. Field of the Invention
The subject disclosure relates to methods and systems for enterprise software application development and management and more particularly, building a suite of collaborative tools to manage the gathering, recording, and dissemination of business and technical requirements, and subsequent conversion to design, and specifications documents for complex systems such as those in the computer software industry.
2. Background of the Related Art
Less than half of all enterprise software projects meet their original time and budget estimates. It has been well documented that the understanding, recording and distribution of requirements and the subsequent proper use of requirements for creating the design and the specifications is at the heart of successfully manufacturing software. There is a perception that requirements are hard to gather. While gathering requirements is not trivial and does require skill and experience, it is certainly not a mystery. Most requirements gathering experts continue to stress the benefits of doing a decent job of requirements gathering.
Many organizations both on the client and vendor side of enterprise software try to create requirements in a somewhat formal way. The requirements are then recorded as text documents using some sort of word processing program. Requirements may also contain schematics and diagrams created in generic drawing tools.
From the moment a Request For Proposal (RFP) or Request For Quote (RFQ) is formally created, various manifestations of requirements from one or more perspectives are being recorded and are available. Some companies also use formal tools, such as Rational Rose, Borland Together, Enterprise Architect, for requirements gathering that often work within a suite of software design tools. Tools built around the Unified Markup Language represent the state-of-the-art in this area.
The most important time for measuring the success of requirements management is the procurement phase, simply because this is the time during which scope, time and budget are being boxed in and estimated. The procurement phase comprises the entire period of time and process from the beginning of RFP/RFQ creation, through the statement of work through the completion and signing of a contractual agreement. At the end of the procurement phase, costs, time and scope estimates have often been formalized, documented, and in the case of fixed-cost-fixed-time engagements, finalized. A degree of transfer engineering already has taken place as the teams assigned to finding a vendor and the team assigned to closing the deal often move off the project and new teams are assigned to the execution of the project.
It is also important to note that the proper gathering of technical and business requirements is a resource intensive activity that often involves technical leads within both vendor and client organizations at a time when nobody is getting paid yet. This was not the case early in the history of the industry, where requirements were taken into the overall budget. Increasing competitive market pressures have made requirements gathering a sales or sales-related or pre-project activity.
In the past, there has been a layer of assumptions guiding the development; namely that requirements are perceived as an aspect of a software project that is managed early in the life of the project. The idea is that the team then starts “building” and like most other human building projects, such as constructing a skyscraper, nothing fundamental will change. Of course, in any project change is inevitable.
The lobby that was destined for carpeting may now require granite tiles, and this is managed with a change order. But a building plan for a two-story shopping mall does not become a 14-story Hospital after the project is started. The idea is that project generally sticks to the requirements without fundamental change. This is the approach and assumption of more stringent methodologies that rely on “phases” that linearly follow one another until the project is complete and the whole process can start again from the beginning for any change or any additions.
In all other fields of manufacturing, change is understood to have impacts on time and money. This is not the case in software. Software is considered ultimately malleable and flexible and so change will have an impact, but that impact may be somehow mitigated or reduced to have only a fraction of the actual manifested cost and time, or so the market perception goes. Frederic Brooks addresses this in his paper entitled No Silver Bullet:                 The software entity is constantly subject to pressures for change. Of course, so are buildings, cars, and computers. But manufactured things are infrequently changed after manufacture; they are superseded by later models, or essential changes are incorporated into later-serial-number copies of the same basic design. Call-backs of automobiles are really quite infrequent; field changes of computers somewhat less so. Both are much less frequent than modifications to fielded software. In part, this is so because the software of a system embodies its function, and the function is the part that most feels the pressures of change. In part it is because software can be changed more easily—it is pure thought-stuff, infinitely malleable. Buildings do in fact get changed, but the high costs of change, understood by all, serve to dampen the whims of the changers.        All successful software gets changed. Two processes are at work. First, as a software product is found to be useful, people try it in new cases at the edge of or beyond the original domain. The pressures for extended function come chiefly from users who like the basic function and invent new uses for it.        Second, successful software survives beyond the normal life of the machine vehicle for which it is first written. If not new computers, then at least new disks, new displays, new printers come along; and the software must be conformed to its new vehicles of opportunity.        
Under current workflows and business realities, requirements are not fixed at some point in time. Requirements constantly change and evolve. Requirements are more effectively thought of the substance of the medium in which a software project is conceived of, created and in which it continues to live. Requirements and their expressions don't go away; requirements and their expressions are not a fixed thing.
Requirements are not just in the software. Requirements surround the software project. For example, the Quality Assurance department is sitting next to the developer one day and the next day, it is being managed out of a place that is 14 time zones and a world of languages away. The documented requirements and the process that worked for the team next door suddenly do not work for the new process. New material has to be added. Previously understood jargon may have to be simplified or rewritten. Change cascades through the process but this new dimension of change is usually not viewed as a part of project requirements. It is just change.
The industry's response to requirements management and the need for their documented manifestations exist in a broad spectrum from the highly formal and rigidly structured, such as the Rational Unified Process at one end, to a more flexible, more lightly-documented, less formal approach such as the Agile software development process.
Rational Unified Process (RUP), when it works well, provides structures and detail but is difficult to manage and maintain throughout the life of a project and is subject to diminishing returns in case of smaller projects.
The AGILE process recognizes that documentation is hard to manage and maintain in the rapidly changing world of software projects, so they do away with as much formal process and documentation as they can, as if doing away with the rigors of requirements management somehow removes complexity. As a result, the AGILE process works well for smaller projects with smaller teams where communications can more easily be managed without the need for a formal process.
What is needed is for the requirements, design, and specifications documents to be structured, richly detailed, and highly specific as in the RUP. But they also need to be flexible, malleable, fluid, and light as in the AGILE process. There is a need for information management tools to address both apparently contradictory needs simultaneously. It is not a battle between one methodology versus another, it is the fact that both are required simultaneously that is at the heart of requirements management and its influence on the fact that less than half of all enterprise software projects in the United States stay within their original cost/time/scope definitions and some 17 percent are actually abandoned.
In other words, one can gather requirements but if you cannot record them, update them, and constantly change them and distribute the effects of the changes to the parties that need to know effectively, efficiently, and quickly, the requirements may not be used in time to influence a layer of decision making. There has been an effective doubling of hardware performance per unit cost every two years now for nearly three decades. This trend is referred to as Moore's Law. No such improvement has ever occurred for software productivity, reliability and cost, despite a myriad of inventions, technologies, improvements, processes, and methodologies over the past three decades.
Computer hardware and computer software are sufficiently similar that an entire field—namely, the field of hardware virtualization—exists based upon this similarity. Computer software has been proven—so far—to be capable of reproducing all aspects of computer hardware except, of course, for Moore's Law.
Frederick Brooks' No Silver Bullet quoted earlier, is the reference description of why software is this way. To see what rate of progress can be expected for software technology, Brooks examines and describes the difficulties of software technology. Software in his generally-referenced descriptions is viewed as follows:                The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract in that such a conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.        
The construct of interlocking concepts is created by a group of people with different skill sets and areas of expertise. This group of people are called the project team. The project team occupies an information ecosystem 80 as shown in FIG. 1A. The information ecosystem 80 is similar to a biological ecosystem in the complexity of relationships and dependencies among those who occupy it. FIG. 1A shows a schematic representation of the information ecosystem. The project team members P0 through Pn are shown directly interacting with the construct of interlocking concepts that is the software entity as defined by Frederick Brooks in No Silver Bullet. The entire system is enclosed by a boundary which defines what we call the information ecosystem. The information ecosystem as defined here in is the sum of the interactions of all the team members and all the information produced, modified and destroyed by them. The net result of and the motivation for the information flow is the construct of interlocking concepts. Nothing is defined as a means of interaction between the team members and with the construct. The interactions and means of interactions are all undefined.
The project team is basically every person that can—and does—modify the information in the ecosystem. The software and its requirements are evaluated in the context of the entire team from the inception of the ecosystem to any time of evaluation. The ecosystem has no end because the ecosystem is a living entity not in a biological sense but in the sense that the ecosystem does not end and may stay around for decades.
In fact a team or member may have contributed a document or specification a decade earlier and no longer be working for either vendor or client. The quality of a decade old document can have a direct impact on a project today. The subject disclosure assumes that the project team is not simply the consumer or producer of the information, but the source of everything in the information system and thus inseparable from it.
The members of the project team however, are not uniform entities. The members have different core competencies, different skills that may or may not overlap in some areas. Each group has a different vocabulary, different graphical conventions, different jargon for referring to their aspect of the software entity. Their dependencies on another member may well be an opaque and derivative dependency from the relationship of other members to that member.
Each member may view the software entity from a completely different perspective. And yet all their requirements intersect somehow, somewhere in the construct of interlocking concepts. A point of intersecting requirements may be a marketing requirement for data display, which may be in complete conflict with the manner in which the data is extracted or presented to the display framework. This conflict is not known at the time of the estimate, but will add significant time to a project and require the hiring of additional unanticipated resources. A simple cost-saving hardware alternative may be ignored early on because of a platform choice that later changed because of a legal compliance issue after the expensive hardware was purchased.
To the Customer Service Manager or marketing specialist at the client, the software entity may be discussed in terms of the interface with which their customers interact and the net result of their successes and failures. To the systems manager at the hosting site, the software entity is a series of hosting servers, routers, and load balancers. To the project manager the software is a series of tasks, subtasks, dates, and resources.
To the software architect designing the system software, the entity is something; to the junior contractor building HTML pages, the software entity is something quite different. Each of these perspectives represents an aspect or components of the interlocking concepts and so is part of the construct. The information contained in this evolving information ecosystem is at once massive and multi-dimensional.
It is accumulated and constructed over a period of weeks, months, and often years, typically by changing teams. During this period, the understanding of the entity itself can evolve significantly. Again to the members of the team creating an RFP, to the presales engineers responding to the RFP and the sales team initiating communications between the parties, the software entity is a series of different but co-dependant other perspectives.
In addition, certainty about the fidelity of the requirements for a software system is gained cumulatively over time. Moreover, the certainty itself is at once highly localized but also widely distributed among the aforementioned team members each of whom may be absolutely certain, or completely in the dark, about any given aspects of the construct. Again because the construct of interlocking concepts is not one thing to one person, but many things to many people, complete certainty about one aspect does not necessarily insure certainty about others.
Although there are definite success stories, today's methods of documenting within office software suites or specialized software design tools does not cope effectively with this multi-dimensional and transient information system in the majority of cases, in part, because the maintenance of the information—when it is actually acknowledged and managed—is transferred from a field expert to someone whose job is documentation and document management. Now all changes have to be transferred through this role and placed in the proper part of the collective document set; when it exists at all, in any effectively maintained and updated state.
The impact of a single change must be examined and documented throughout the system and absorbed in its totality in order to capture a small overlapping aspect of the change. While this, occurs during the designated requirements phase early on in the project for changing requirements, it is not maintained as thoroughly once the design and implementation phase is started. This is in part because of resource allocation. The specialist in charge at the beginning, rarely if ever, stays throughout the project to manage change within requirements, and specifications. It is left up to the team members to maintain any necessary documentation, in addition to performing their regular tasks within the project. Since this can be a time-consuming activity, it is rarely, if ever carried out in practice, since time for “continuing to understand and document” the project is rarely billable after the designated requirements phase early on in the project.
A verbal description of the various states, even when supported by diagrams may be able to contain the complexity, but it would require a very large and difficult to produce document. Even if said document was produced, it would have to contain things as diverse as Data descriptions, user interface artifacts, project plans, and architectural schemas among others. It would then have to contain information regarding all the relationships between the various states and aspects. Furthermore, said document would be cumbersome and difficult to maintain from all the perspectives, search, navigate, access and view in each individual context. No textual description can contain such rich, multi-layered, multi-faceted information and yet be easy maintain and to navigate and to readily show dependencies; only a simulation can.
The Unified Markup Language (UML) is one attempt at containing this rich and multi-layered information system. although the UML's main goal is modeling and building object-oriented software and not managing information. One shortcoming of the UML is the expertise required in learning and using it. This expertise is in addition to any software expertise the audience may already have. In other words, one may be an expert in an area of software design, but be unable to discuss that area with an expert from another of software's disciplines without both of them first having to become UML experts. This of course assumes that each area of expertise is well-contained and handled within the UML's syntax and semantic framework which in reality is not always the case as Klaus-Dieter Schewe describes in his paper UML: A Modern Dinosaur” [4].
Learning UML is at least as difficult as learning any other language, even if each area of expertise was well-covered in the UML. This learning curve presents a barrier to entry for team members trying to define their perspective: all team members must fluently speak, write, and draw in UML. Even if this was not a significant shortcoming for any tool attempting to overcome this communication problem, it would add a layer of complexity to already complex software systems. According to Brooks, software systems are “arguably more complex for their size than perhaps any other human construct, because no two parts are alike (at least above the statement level).”
Brooks continues the statement about the “construct of interlocking concepts” thus:                I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. [ . . . ] If this is true, building software will always be hard. There is inherently no silver bullet. [3]In other words, it's harder to specify and design software and test the fidelity of the design and specification than it is to actually build the software.        
Conventionally, the most thorough description of a piece of software is source code. Only the software source code itself describes the behavior of software with complete accuracy. However, software source code as a descriptive form has several limitations.
First, Software source code consists of instructions rather than descriptions. Since the primary purpose of source code is to dictate specific behavior to a machine, the primary goal of all source code is to describe behavior in a manner that can be easily broken down into simple atomic instructions. As a medium, software source code is very good at communicating the ‘imperative’, but it is limited when trying to communicate the ‘descriptive’ or the ‘inquisitive’.
Recording the descriptions, questions, and answers that are important to humans but have nothing to do with ‘instructions’ is difficult in software source code. Descriptions of purpose (“why was this code written”), intention (“what is this code supposed to be doing”), explorations of possibilities (“this code might be used to . . . ”) are all important to humans, but not to a compiler. Such descriptions can only be contained in source code in the unstructured and arbitrary format of ‘comments’. When source code is compiled, the descriptions that do exist in comments are completely removed from the resulting ‘compiled’ code. This limitation is well recognized within the industry and attempts have been made to address it.
Second, Software source code is very difficult for non-experts to understand. Software ‘illiteracy’ is a particularly difficult problem to overcome. For most forms of communication through language, a distinction is drawn between the ability to speak a language, and the ability to read and write in that language (‘illiteracy’). In many English-speaking countries, a man can obtain a license to drive an automobile without being able to spell the word ‘AUTOMOBILE’; he may even be able to earn a living teaching others how to drive. Conversely, in order to have a meaningful conversation about software source code, one must be able to read and write in computer languages. A COBOL programmer may discuss software with a JAVA programmer, a BASIC programmer may discuss software with a C# programmer, but the non-programmers in the room cannot participate. This is another limitation of software source code as a general-purpose communications medium. Attempts have been made to simplify the presentation of software source code, from flowcharts to the Unified Markup Language to graphical programming languages. But these are again highly complex languages that require not only a knowledge of programming, but the additional of need for acquired expertise in the languages themselves, adding complexity instead of alleviating it.
Third, Software source code assumes absolute contextual understanding. Source code is extremely difficult for even experts to read, because the meaning (intended atomic instructions) of software source code changes from system to system. The most difficult part of programming is determining what the actual behavior of a line of code will be once it is executed in the “production” environment. Software experts are often surprised by the actual behavior of software source code “in the field”. Modern production computing environments are highly advanced. Layer upon layer of code have been built up to comprise the current software deployment environments. Some of this code is inappropriate to the current context. For example, architectural decisions made to accommodate peculiarities of certain token-ring networks in the early 1980s are still present in modern networking APIs, such as those that a modern cell phone uses to send and receive email. Contextual dependency is the most difficult barrier to software source code as an effective general communications medium: “ILLEGAL OPERATION 0x876” means nothing out of context. On a fundamental level, the code that describes an interface and the code that enables communication between two different subsystems are essentially of the same form. Their external manifestation is therefore opaque except from those who can decipher the code.
Being able to design, specify and test a complete prototype during procurement would of course be the ideal solution, but real prototypes cost money and resources and neither vendor nor client is likely to put together anything better than a high-level demonstration at best. Thus, a need exists for a practical approach to successfully develop complex systems.
Several approaches have been well documented. For example, see U.S. Pat. No. 7,114,149 issued on Sep. 26, 2006 to Aptus et al., which discloses a software development tool that generates HTML documentation for source code in a software project. The HTML documentation includes graphical and textual documentation. The software development tool utilizes the linking capabilities of HTML to provide navigation links between the graphical and textual documentation, thus facilitating navigation through and viewing of the documentation. The tool of Aptus et al. merely generates documentation that is an electronic copy of the traditional specification document. The links are typical user-selectable items that bring the user to another page. If the source code is updated, the HTML document is updated but not vice versa.
Another approach is shown in U.S. Pat. No. 6,789,054 issued on Sep. 7, 2004 to Makhlouf. Makhlouf discloses Adaptive Model-Reference (AMR) tools that visually specify the integrated architecture model of a real system, assess its value, and automate the static and dynamic binding of its model components into adaptive systems. These systems can be adapted during system development to compensate for requirement changes and design errors, and during run time operation to compensate for unanticipated operational system conditions. Makhlouf does not address how non-technical audiences can effectively communicate in his system.